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As an evaluator and an educational researcher into the processes of teaching 
and learning in post-secondary education, I am fascinated by the array of procedures 
that can be used to describe and analyze the learning process. Many of the proce- 
dures that we have encountered in our studies of university courses and programs 
have yet to be documented. Most studies of the learning process in post-secondary ' 
education suggest a variety of methods without recognizing categories, central 
tendencies, or prototypes. Disciplinary boundaries are barriers to understanding, but 
they are minor compared to the barrier created by the lack of a vocabulary to 
describe the learning process in consistent, explicit and operational terms. 

For the past nine years at McGill University our program of research into the • 
nature of the learning task has delved into crannies and explored the peaks of the 
knowledge territory. At times we have slogged through the swamps and at others we 
have been terrified of precipices and crevices. One abyss has been the concept of 
relativity, which I have come to the conclusion was developed as a limit on the 
concept of determination in physics. But, while in physics, understanding was 
suffering from hardening of the categories, educational theory appears to be suffer- 
ing from a lack of categories. In the field of educational theory, we have not yet 
developed models which would provide us with explanatory structure, yet we are 
prey to a Zeitgeist which questions ihe possibility of concrete knowledge and 
grounded theory. The path through the knowledge terrain is not an easy one and, 
though the land is by no means a desert, there appear at times to be considerable 
cactus and bramble underfoot. 

My intent in this paper is, then, to describe and to illustrate certain methods 
that have been tested in our program of research on the learning task in the univer- 
sity. The methods are taken from this research with comments added from the 
more general scene of educational research in Canada, The methods have been 



selected to Illustrate four phases of inquiry: conceptualization or nnodel-building; data 
gathering techniques; the representation of data; and the analysis of data which 
focuses on the measures used to assure that the research effort has been meaning- 
ful. The methods have aspects of both qualitative and quantitative approaches to 
research but what is most important is the recognition of criteria to judge their 
effectiveness. 

Conceptualizing the learning task 

What does conceptualization or model-building have to do with methods of 
research? One of the early findings of the Standing Committee for the Promotion of 
Research of the Canadian Society for the Study of Education concerns how research 
proposals are conceptualized. The committee was told that many grant proposals in 
educational research lack a clear presentation of the objectives of the research and 
why the proposed research is important. We do not put our research in context 
and explain it to others. During the past three years I have been a member of a 
committee which reviews some 120 educational research proposals annually in Que- 
bec. The criteria for success are clearly stated in the guide for submitting proposals 
and the procedures for evaluating the proposals are judiciously followed. The first 
characteristic on which proposals are judged is the problematique or conceptualiza- 
tion of the problem. The problematique sets the organizational structure which will 
guide and govern the entire project, from data gathering to conclusions. Too often 
the decision of the grants committee is that the problematique does not justify 
doing the research. This may be because the question has not been posed in a suffi- 
ciently explicit manner, because the research literature has not been reviewed, or 
because the would-be researcher has neglected to state what problems would be 
solved by the investigation. 

Why do we not pay more attention to conceptualization or model-building? I 
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would hazard the answer that our profession places greater faith in clinical readiness 
than in analytic procedures: on the way to learning to survive in the highly connplex 
nnilieu of education we have developed perspectives and strategies that do not honor 
deductive logic. Yet it is logical structure that is the hallnnark of good research. 
Another problem that has limited educational research is that classical approaches to 
research design, borrowed from the natural sciences, allow the researcher to look at 
very few variables at one time and thus do not fit our multivariate circumstances. 
The computer and anthropological and sociological methodology have changed that. 
It is now possible, with the help of Cook & Campbell (1979), Cuba & Lincoln (1981), 
Miles & Huberman (1984) and many others, to create a comprehensive design of 
what is to be examined. I do not wish to imply that the methods developed for the 
study of learning by psychologists have thus become invalidated. To the contrary, for 
methods of studying learning we in education will remain indebted to the discipline 
of psychology. An obvious corollary to this is that educational researchers should 
maintain a watch on the methods that are being used in our basic sciences, the 
social sciences. 

To illustrate the development of a conceptual framework, let me describe how 
the model of the learning task project at McGill University began and changed. In 
the beginning, we were looking for a unit of analysis, or measurement, and decided 
that the simplest and most evident one was the concept, a unit of thinking, which 
could be found in instructional material and which was limited in size (from one 
word to a phrase) and in form (written). Our work was primarily descriptive: we 
wanted to determine what concepts professors considered to be important in individ- 
ual courses that they taught, and we wanted to know what methods best described 
these concepts. The methods that we tested were fron the psychological literature 
and included cardsorting for similiarity grouping, cluster analysis, word association. 
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tree structuring and sinnllarlty rating (Donald, I983K 

After spending three years establishing the domain and the nnethods most useful 
to describe It, we then spent the next three testing ti^e extent to which concepts 
could be used to describe student learning* We found, among other things, that 
concept knowledge predicted course achievement better In the social sciences than 
grade point average, although It did not predict better in the physical and biological 
sciences or In the humanities (Donald, I984K We still did not understand, however, 
the role of concepts In the organization of a course* We therefore spent another 
two years doing a semantic analysis of the concepts* This analysis showed us the 
basic elements and relationships that occurred among the key concepts in a course, 
and gave us a means of showing the organizing principles In the course. In a con- 
current project, we focused on the next larger unit of analysis, the proposition, and 
studied Its function in a course. We found that by moving from the concept as unit 
of analysis to the proposition, we had complicated our problem threefold, but we 
also added to our understanding of the knowledge structures in a course (Donald & 
Nagy, 1985). 

Doing this research we worked closely with professors. In our Interviews with 
them and from our attempts to measure the usefulness of concepts and propositions 
in the evaluation of learning, we began to see the need for a shift in focus. We 
had been looking at knowledge structures, but when we asked the professors what 
they considered important for students to learn, they replied in terms of the skills, 
such as analysis and synthesis, that they would like their students to acquire, A 
literature search of the skills considered Important in post^secondary education, 
including critical thinking, problem solving and cognitive processes led to the 
development of a model of Intellectual skills which could then be tested in univer- 
sity courses (Donald, 1985). We thus found ourselves with a second, complementary 
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model of the learning task in post-secondary education. But how did intellectual 
skills relate to the knowledge structures in a course? Do certain concepts or propo- 
sitions imply certain skills? Are particular concepts or propositions embedded in the 
skills? 

To answer these questions required a further shift in focus, ro the broader 
question of what constitutes a discipline, because knowledge structures and intellec- 
tual skills interact within and are determined by discipline (Donald, 1986a), At this 
point we were in the realm of philosophy. Philosophers investigating knowledge 
suggest that different disciplines have different logical structures in their concepts 
and propositions, different criteria for judging the truth of propositions, and different 
methods for producing those propositions (Adier, 1982; Hirst, 1974; Scheffler, 1965). 
These characteristics obviously would have to be token into account in a conceptual- 
ization of the learning task. 

Study of the logical structure of disciplines appeared by definition to consist of 
finding the organization of data or ideas showing the relationships between compo- 
nent parts. Our research on the relationships between concepts in courses 
provided a starting point for looking at kinds of structure. We knew for 
example, that sixty percent of the relationships in our study of key concepts were 
similarity relationships and that forly percent were dependency (procedural, logical 
or causal) relationships. Recent literature in cognitive science discriminates between 
well and less well structured disciplines (Frederiksen, 1984). In addition to 
differences of degree and kind of logical structure, we could expect level of 
abstraction and generality to affect logicoi structure. We hod found, for example, that 
key concepts in the social science courses were more abstract than those in the 
physical sciences. The relationship between logical structure and truth criteria 
appears to be intimate, since truth criteria are usually measures of the coherence. 
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consistency and precision of logical structures, whether established on the basis of 
experience or agreement. We could suppose that truth criteria would picy a particu- 
lar role in the intellectual skill of verification, which was the skill that professors in 
our study of intellectual skills had declared was most important for expertise in a 
discipline. 

The conceptual framework had thus been elaborated into a multidimensional 
model with four interacting sets of dimensions. Figure I shows the interaction 
between knowledge structures and intellectual skills, with the effect of logical 
structures and truth criteria on them. The conceptual framework has evident 
weaknesses. It does not show, for example, the immediate relationship of the truth 
criteria to verification skills, nor does it show how logical structure is found in 
the propositions. It does, however, make us cognizant that all of the dimensions are 
inter-related and that all of them play a part in defining the learning task. From 
this framework; for any one study, the questions which linked certain dimensions 
would determine which relationships would be concentrated on for data collection 
purposes. 

Figure I here 

For the purpose of conceptualizing a problem, my first objective in 
this paper, the figure provides an example of c representation of the context and 
scope of a problem. It is important to note that in the development of this 
framework, we moved from a focus on one kind of knowledge structure to two 
kinds, from a set to study knowledge structures to a set to study intellectual skills, 
and from questions about the relationship of knowledge structures and intellectual 
skills, to the encompassing question of what constitutes a discipline. Each step 
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required investigation of the literature in the area, and each step was clarified by the 
collection and analysis of data. The innportance of conceptualizing a problem by 
representing the connponent parts lies in its ability to tell nne what I can expect 
to learn and what questions I nnust ask. It also forewarns me of the difficult or 
fuzzy areas that lie ahead. 
Data collection 

Collecting information about learning could be expected to be a straightforward 
process, since it is a necessitatum in education. Whether it consists of collecting and 
grading assignments or preparing school leaving certificates, information has been 
collected so regularly that it is background in the educational endeavor. The use of 
standardized tests is routine. At the post-secondary level, however, little agreement 
has been reached on measures of learning or achievement, and the current movement 
in the United States to ensure academic competencies through testing can be pre- 
dicted to have a rough road ahead of it. A review of the conceptual framework in 
Figure I shows why : we do not yet have a sufficient understanding of how knowl- 
edge structures are developed and what level of consistency can be expected from 
them. We do not know what role intellectual skills play and how they are related to 
knowledge structures. We cannot yet pinpoint the logical structure of a unit of 
instruction, and we do not know how logical structure and truth criteria apply to the 
knowledge and intellectual skills that make up the subject matter. Add to this lack 
of knowledge the known inconsistency between experts in a subject matter area, as 
witnessed in the Goldmen, Schoner & Pentony (1980) study of concepts considered 
important in the field of political science. Out of a concept inventory of just under 
22,000 terms, 10 experts could agree on only eleven terms as being most important 
for undergraduate knowledge of political science. If we accept the principle that 
disciplines are continually developing, then data collection on the learning task 
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requires Interactive qualitative methods that can be used to measure both consis- 
tency and rate of development over time. 

This is not to say that quantitative methods are not useful. In our study of 
student learning of concepts we had students define the key concepts, and we then 
scored them and performed multivariate analyses on the variables affecting learning. 
The results were worth the trouble in one final year course in the sciences where 
the sources of extraneous variability were at a minimum. What 1 am suggesting is 
that, given the state of our knowledge about the variables affecting learning, and 
the nature of the learning task itself, a great deal of frankly exploratory work has 
to be done before we can achieve the level of conceptual precision which merits the 
use of quantitative methods. 

Given these limitations, we have been investigating the use of semi-structured 
interviews with review by a third party and confirmation by the interviewee of the 
report based on the interview. The step of producing and reviewing a report of 
the interview is the closest we have come to a reliability check, and even though it 
requires a great deal of labor to produce it, it appears to be integral to the process 
in terms of maintaining both our records and the support of the professors who work 
with us. The interview structure is developed from the questions that arise in the 
conceptual framework of the project. We have learned to go to an interview 
equipped with a model of what we want to know, and to be prepared to come away 
with the professor's perspective of what we should want to know. Flexibility in 
our approach to interviewees is essential. Some professors require time to explain 
their own perspective before they will look at our conceptualization. Some require 
that the conceptual model be left with them so that they can examine it and for- 
mulate their responses over time. Some reject the notion that their subject matter 
can be described. We have to be both elastic and persistent over a period as long as 
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a semester, and we rrjst expect to be open to a broad spectrunn of Ideas of what 
constitutes the learning task. Each professor has a perspective on learning, some of 
which ore quite elaborate, and the perspectives provide tests of the consistency 
and scope of our model. 

Why, then, go in with a structure at all? There are tv-o reasons that our expe- 
rience would support. First, without a structure we would be in danger of informa- 
tion overload from the wealth of experience each professor possesses on the subject 
matter of teaching. Second, we have found in our research that the framework that 
professors have of what constitutes teaching may be far removed from the frame- 
work we are investigating. We discovered this several years ago when we inter- 
viewed professors about their evaluation techniques and found that they, instead of 
talking about how they evaluated learning, talked about the instructional process. 
The interviewer, therefore, in addition to being flexible and persistent, must be able 
to reroute the conversation back to the specific interview questions. A final 
criterion of the successful interview is that it be enjoyable and worthwhile to the 
professor. We have interviewed students and reimbursed them for their time spent, 
but the time of the professor is both much more limited and nonreimbursable. It is 
therefore important that the professor come to the end of an interview v/ith positive 
affect, feeling that he or she has made a contribution and that it has been appreci- 
ated. In the written report that we ask the professor to verify, we express our 
appreciation of their contribution and invite their advice. 

What form should such an interview structure take? We operate with the 
expectation that the professor's time is limited and valuable and we therefore 
proceed immediately to a description of what kinds of information we would like 
them to give us. We provide them with a copy of our conceptual model, inviting 
them to become co-researchers with us, and we also provide them with the set of- 
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questions we need answers to. There is no requirennent that the questions be ans- 
wered linearly, that is, without returning to previous questions. One of the inter- 
viewer's responsibilities is, however, to record the interview on a taperecorder so 
that the answers can be located with reference to the questions. At the same time 
the interviewer is expected to explain and to work with the professor to elucidate 
the model and provide examples of it. Over the years I have found that senior 
undergraduate students who have a good sense of the university context make superb 
interviewers, and 1 have found that they are more doggedly persistent and can use a 
naive strategy which supposes nothing and therefore garners more specific informa- 
tion in answer to the questions than I could. We try to do interviews in parallel, 
that is, with two professors from an area of specialization, in order to increase our 
understanding of what is essential to the domain, and we have two editors from the 
research project producing each report to ensure its readability and meaning. 

This data collection procedure, cumbersome though it may appear, allows us to 
go beyond our framework while getting answers to the questions we intended, and it 
enlists the aid of the people who are most important to our project, the subject 
matter experts. They are invariably willing to discuss and to continue working with 
us, and some have stayed with our project for over eight years now. This is an 
essential resource if we are to understand the learning task. 
Representation of the data 

A few years ago, T'jkey (1977), in the book Exploratory data analysis. 
suggested that data collected deserved a careful perusal before being subjected 
to statistical analysis. His methods of descriptive analysis of data into, for 
example, stems and leaves, revealed form and pattern, that is the spatial organiza- 
tion of the data, which allowed us to understand our data graphically. Just as a 
conceptual framework allows us to see what theoretical relationships we should be 
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Tho second form of graphic representation which we applied to our data in the 
cognitive stnicfures project wos o form of graphic analysis developed by Woern 
(1972), which did not require o stepwise linlcoge but instead used a rating of degree 
of roiotionihip which wos tbmx transferred to poper to show equolly strongly related 
coneeptto The Woern graphic onoiysis produced o parallel form of representation 
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which could than be compared with the tree structure for the same set of concepts* 
The consistencies between the two representations were proof of the relationships 
between concepts, but the differences were also revealing. For example, concepts 
which were loosely linked In the philosophy course tree structure showed much more 
extensive relationships In the graphic analysis (Figures 2 & 3). Groupings of concepts 
such as virtue. moralltVt morals and duty were more clearly visible* Thus the 
graphic representations suggested hypotheses about the concepts which could then be 
followed up In o interview with the professor* 

Figures 2 & 3 here 

A third form of graphic representation which we developed in our study of the 
key concepts from selected courses built In the logical structure of the concept. In 
order to show how elements of a concept contributed to its meaning, we had to 
introduce rules which would soy whether one element of the concept was hierarchi- 
cally related to another or whether one element affected another. For example, 
behavior is changed through a reinforcing agent (Figure 4). This is represented by an 
arrow from the agent to the behavior. The act of representing the concept required 
the specification of rules which could then be used to understand the relationships 
engendered by the pattern. 

Figure 4 here 

Each form of graphic representation allowed us to become more explicit about 
the relationships between elements, and suggested the pattern or Organization of the 
important concepts in a course or the individual concepts themselves. The represen- 
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tations not only provoked discussion about the structure of knowledge, but 
gave direction for instructional paths to be followed or avoided. For exannple 
the philosophy course representations would suggest that good and virtue are pivot 
concepts and should be focussed on as the conceptual foundation of the course. 

Figure 5 here 

A nnore classic fornn of representation which we used to compare the descriptive 
elennents and relations of key concepts in a course was the matrix. In the psychol- 
ogy course on Thinking, for example, the representation of the common concepts and 
elements showed us that internal models of the world, cognitive structures, and 
schema were composed of rhe same elements and had, therefore, a high degree of 
similarity (Figure 5). In our recent research on the intellectual skills developed in 
university courses, we have been using spread sheets to show the use of skills in 
particular courses, their development, evaluation, and their importance for expertise 
in the discipline. The spread sheet allows one to compare different elements in a 
model for their utilization in a particular situation. For example, in the educational 
research course, on entry to the course students were expected to be able to use 
only one of the selection skills, identify critical elements, out of four (Figure 6). 
The other three skills, choosing relevant information, ordering information in 
importance, and identifying critical relations were to be developed in the course. The 
representations frame the data base, show scope and limitations, identify the orga- 
nizing principles, arrange the data into a gestalt, and allow us to verify the 
components and the connections between them. Evaluation research has developed 
methods and models of illustrating data in order to explain complex findings to a lay 
audience (Patton, 1981; Stake, 1975). As educational research increasingly deals with 
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more complex variables and relationships, representation becomes a crucial step. It is 
an essential precursor to making inferences or synthesizing. It is expected that 
tables and figures will be used to describe statistical, that is, inferential and 
verification procedures; I am suggesting that we begin to think graphically two steps 
before. 

Figure 6 here 

Analysis of the data 

The greatest variability occurs in educational research in the degree to which 
and the methods by which data .are analyzed. With backgrounds in such diverse dis- 
ciplines as history and mathematics, educators have a confusing variety of 
approaches to choose from. A variety of criteria for judging the validity of results, 
from coherence to 3tatistical significance also exists. Some would say that the 
variety is sufficiently great to impugn our credibility as scholars. With this 
richness of possibilities, how is educational research to be authenticated? An 
example of the data analysis procedures which we followed to test a model of 
intellectual development in post-secondary students became a search tor criteria of 
validity. (Bateman & Donald, 1986). 

The model. Perry's scheme of intellectual development in the college years, 
posits four broad stages or positions which students go through (Perry, 1970). In the 
first stage, dualism , the student views the world and knowledge in absolute terms: 
things are either right or wrong, and the teacher knows the correct answer. In the 
second stage, multiplicity, the student begins to recognize the authorities' contrasting 
viewpoints but sees contrasts as the authorities' way of making the student locate 
the right answer. Students in the third stage, relativism , are aware that there are 
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no right or wrong answers. They recognize the need to analyze and evaluate. In the 
final stage, connnnitnnent, students have developed their own point of view, recognize 
that it is their own, and are able to act according to it. The question that we 
asked was whether this nnodel did indeed describe student developnnent. 

But how could we test this? Since we had four stages of development post- 
ulated, one test would be to see if different nneasures of each stage measured the 
same thing, that is, response to different statements describing each stage should 
converge. According to Cook and Campbell (1979), convergence of measures within 
a stage and divergence between stages would be proof of construct validity. We also 
were aware of research done on the construct validity of measures of cognitive 
structure by Shavelson & Stanton (1975). 

A 16 item questionnaire was developed from items used in the research litera- 
ture to describe the four stages and was administered to students in their first and 
second years of college. To test for convergence within stages and divergence 
between, a correlation matrix was computed between all items. If the model held, we 
would expect significant correlations within stage and non-significant correlations 
between the four stages of dualism, multiplicity, relativism, and commitment. We 
found 67% significant correlations for the four commitment items and 50% for the 
items from the other three levels. Convergence was therefore not great. Outside the 
stages, however, the percent of significant correlations was 28%. Thus we had 
greater convergence in than out. Further exploration showed that few of the signif- 
icant correlations occurred between dualism items and others (10%), and that the 
percent of significant correlations among the items describing the more advanced 
levels of intellectual development (46%) closely approximated the percent of signifi- 
cant correlations among items within three of the groups. This suggested that 
dualism items were not related to those at other levels, but that the three more 
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advanced stages were somewhat related. 

More intensive study of the Items based on their correlations suggested that 
other factors might be better able to explain student development. For example, 
three of the four statements describing dualism referred to knowledge as the accum- 
ulation of facts. Three relativism Items had the theme of the development of an 
individual point of view. A principle factor orthogonal varimax analysis was there- 
fore done to reduce the 16X16 correlation matrix and yielded five factors. The 
first three factors were students' responsibility for their own learning, using 
evidence, and recognizing one's own point of view, and each factor had its highest 
loadings from items from the three more advanced stages. The fourth and fifth 
factors, finding the right answers and the teacher's role as giver of knowledge, 
encompassed the four dualism items. Thus instead of four stages, we had found five 
factors. A second factor analysis was performed to test divergence more directly 
among the four stages. To do this the scores on the four items in each level were 
combined by student. Only one factor, dualism, was extracted. There was no evi- 
dence of divergence among multiplicity, relativity, and commitment itemb. We could 
conclude, therefore, that rather than four stages of development. Perry had found 
statements which describe two broad levels of intellectual development. 

These procedures established that Perry's scheme had limited construct validity, 
but what would tests of the scheme against other measures of student development 
reveal? What kind of empirical validity could be expected of the scheme? To test 
this, the responses of first and second year students were coiTipared, and students' 
responses were regressed on their cumulative averages. First and second year stu- 
dents did not show a different response pattern, suggesting that if a change occurs, 
it does not occur between the end of the first year in college and the second. 
Significant correlations were found however, between students' cumulative averages 
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anddualism scores (negative), and comnnitnnent scores (positive). Thus, although a 
year in college did not affect responses the questionnaire, student ability did. 
Again, sonne support was shown for two broad levels of developnnent. 

The analysis necessitated a nnajor change in the way we thought about this 
nnodel of intellectual developnnent. The account has been sinnplified to spare you 
the gnashing of teeth and grinding of connputer results. Given our results, we ask 
how nnany nnodels being used to describe educational processes have been checked 
for validity? We ask ourselves if the validation process is always this difficult? We 
know that we nnust continufj to do this kind of exploratory analysis of results, then 
follow it up with confirnnatory analysis. Doing this kind of analysis is both compel- 
ling and dangerous. We sought only to confirm a way of describing student develop- 
ment; we ended up with a rival paradigm. 

These methods used to study the learning task have certain aspects in common. 
They are exploratory. They involve a conceptualization which is then represented. 
They require logical justification procedures and criteria. There Is uncertainty in all 
of them. They make a statement to the effect that they are all conceptualizations, 
not the ultimate truth. But they can be tested, and they can be replicated. They 
can be examined for the number and kind of instances in which they hold and for 
their precision and coherence. They do not confirm, but they challenge. They are 
time consuming, but they are creative, one could almost say aesthetic, endeavors. 
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